Skip to content

solver: re-send the signed stage 1 instead of signing a second one in the same round - #178

Merged
EnzinoBB merged 5 commits into
masterfrom
fix/stage1-resend
Nov 9, 2026
Merged

EnzinoBB merged 5 commits into
masterfrom
fix/stage1-resend

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

For the point release community.18.1.

What happens today

Sometimes a confidant receives a second round package for the round it is in, with more real trusted signatures than the first. It then replaces the first package and enters stage 1 again (SolverCore::nextRound(true)). It signs a second stage 1 for the same round and sub-round, with a new timestamp. As a result, the other confidants can hold different stage-1 copies of that seat. A receiver keeps the later timestamp only if it gets both copies. The confidants then compute different block timestamps, or see two hash-signature pairs for the seat at stage 2. Either way, the seat or the block can fail the stage-3 check.

On mainnet, the community.18 diagnostic lines on one operator's node showed exactly this:

  • the node's round package confirmations, version and roundCost were identical to those in the network's block;
  • the network block's timestamp can be recomputed to the millisecond from the 6 other confidants' stage-1 timestamps, as that node held them;
  • the only input that differed at iteration 0 was the node's own stage-1 timestamp. It had signed stage 1 twice in that round.

The change

  • Diagnostics (commit 1). The stage 1 this node signs is kept, together with the round, sub-round and seat key it was signed for (SolverCore::sentStageOne_). The line Current Roundpackage of N will be replaced by new one moves from debug to info level. When stage 1 is signed again, an info line gives both timestamps and the content fields that differ.
  • Fix (commit 2). When this node already signed a stage 1 for the same round, sub-round and seat, it sends the stored signed message again, byte for byte (stage1 resent unchanged). That message may lack the trusted candidates and packet hashes that came in later. These fields are only this node's votes for the next round, and the other confidants already hold and accept the stored message. A new stage 1 is still signed (stage1 signed again ..., reason <r>) when the stored one would be refused or would make the seat look faulty (cs::stageOneResend):
    • hash-changed: the characteristic hash changed. It must match the majority's.
    • not-after-last-block: the stored timestamp is not after the last block's, so receivers would refuse it.
    • too-old: the stored timestamp is more than DefaultTimeStampRange (30 s) old. This node's own stage-2 check measures the others' timestamps against its own, so their fresh timestamps could fall outside that window. This condition was not in the experiment. It cannot apply to a replacement that arrives within a normal round, and when it does apply it restores the previous behaviour.
    • unreadable: the stored timestamp cannot be read as a number.
  • Lifetime of the stored stage 1. nextRound(false) clears it, and so does the start of every bootstrap round: bootstrap tables also start rounds with updateRound = true, so the code checks isBootstrapRound(). sent_stage1() also checks the round, sub-round, seat index and the key at that index.
  • Docs. A new section 5 in docs/consensus-diagnostics.md, and a row in docs/CHANGES.md.

Cases that still sign a second stage 1

In two cases the node still signs a second stage 1 for the same round, as before. These will be a follow-up:

  • (a) Writer-side replacement. The moreTrusted branch of Node::getRoundTable handles a node that holds its own round package. That path starts the round again with updateRound = false, which clears the stored stage 1.
  • (b) Replacement inside a bootstrap round. A fuller package that replaces the first one inside a bootstrap round finds the stored stage 1 already cleared, because the start of a bootstrap round clears it.

What does not change

  • Receivers. Only what this node sends changes. The resent message carries the original signature over (round, sub-round, message hash), and Node::getStageOne verifies it exactly as before. A receiver that already holds it ignores it in SolverCore::gotStageOne, because the timestamp is the same.
  • stageOneStorage. Its semantics are unchanged. It is still cleared by every nextRound, and the node's own resent stage 1 enters it through add_stage1 like any other. The node's own view of its seat is now the same as the view the other confidants hold.
  • Normal rounds. The pending fix 2 of the experiment (the candidacy pause) is not included.

fatal ConveyerBase> Meta round currently in conveyer

The experiment logged this line once per replay. It comes from Conveyer::setTable, which performRoundPackage calls on every same-round replacement. The conveyer already holds that round's meta, so it keeps it and only logs the line. The line is older than this change: it dates from 165d17b and is in official 4a95aaa. It appears in the baseline-hook run too: 12 lines for 12 replays, against 51 lines for 51 replays with the fix. Its level is fatal, but it is only a log line and the node keeps running. In the hook experiment it comes from the hook's replay, which takes the same path as a natural replacement. It is neither caused nor changed by this PR.

Tests

  • Unit (tests/csnode/stageoneresend_tests.cpp, 9 tests):

    • each branch of the resend decision, and that the hash check comes first;
    • the content diff, with candidates compared as a set, and the reason names;
    • the resent bytes verify, with the receiver's check, in the same round and sub-round, and fail in any other;
    • the decoded timestamp equals the first one.

    The local nodetests run passed all 507 tests before the review follow-ups. After them, the 12 StageOne* tests pass.

  • Private networks: the results are in the comments below.

🤖 Generated with Claude Code

EnzinoBB and others added 4 commits November 9, 2026 17:30
…cement; log a re-signed stage 1 (diagnostics)

The stage 1 this node signs is kept with its round, sub-round and seat key and survives
nextRound(true). When stage 1 is signed again for the same round and sub-round, an info
line shows both timestamps and which content fields differ. The replacement of a round
package by a fuller one is logged at info level. Nothing sent or computed changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the same round

After a fuller round package of the same round replaced the first one, a confidant enters
stage 1 again (SolverCore::nextRound(true)) and used to sign a second stage 1 with a new
timestamp. The confidants could then hold different stage-1 copies of that seat, and so
compute different block timestamps or see two hash-signature pairs of the seat at stage 2.

Now, when this node already signed a stage 1 for the same round, sub-round and seat, it sends
the stored signed message again, byte for byte. Its trusted candidates and packet hashes may
lack the ones that came in later; these are only this node's votes for the next round. A new
stage 1 is still signed when the stored one would be refused or would make the seat look
faulty (cs::stageOneResend): the characteristic hash changed, the stored timestamp is not after
the last block's, or it is more than DefaultTimeStampRange old (the own stage-2 window is
measured from the own stamp).

Only what this node sends changes; receivers verify the message as any stage 1. The stored
stage 1 is cleared by nextRound(false) and at the start of every bootstrap round, and
sent_stage1() still checks round, sub-round and seat key. The "stage1 signed again" line now
gives the reason; the content comparison moved to cs::stageOneContentDiff. Unit tests cover
the decision and that the resent bytes verify at a receiver in the same round and sub-round
only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB

EnzinoBB commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Private-network results

All networks: 7 nodes; the even nodes run the official node (node_alt_ref=official); contracts on; target sequence 1500 (1100 for testnet.yml).

Fault and replay scenarios. These ran in Robustness run 42663079091. It was dispatched from the scratch branch exp/stage1-resend-hook, which holds this PR, the experiment's CI commits and the test hook. That branch is not to be merged.

  • The delay fault is 400 ± 200 ms on n3 and n4. The reorder fault is 25 % on n3 and n4.
  • The replay hook is active in the s1r-* scenarios. Node n5 processes its round package again 1 ms after its stage 1, in every 5th round, as a same-round replacement does.

Column meanings.

  • stage-1/2 marks: marked as untrusted (bad signature) and (sent bad hash-signature pair ...) lines on all nodes. These are the marks a second stage-1 signature causes.
  • other marks: the remaining untrusted marks. They come from the fault itself: no candidates - probably silent, and the smart-contract consensus (Smart: Confidant ... wrong signature / silent / fee-vector). They also appear without the hook.
  • stamp splits: (round, seat) pairs for which the nodes held different stage-1 timestamps.
  • re-entries: how n5's stage 1 was sent again (resent unchanged or signed again).
  • excluded after: re-entries after which n5's own key was gray-listed.
scenario result stage-1/2 marks other marks stamp splits re-entries excluded after
s1-delay (no hook) pass 0 4 silent 0 – –
s1-reorder (no hook) pass 0 18 smart 0 – –
s1r-delay pass 0 14 smart 0 2 resent 2 (R5, R25)
s1r-reorder pass 0 0 0 21 resent 3 (R5, R480, R1535)
s1r-resign (fallback forced) pass 12 (8 bad signature on n5, 4 bad pair) 14 smart 0 3 signed again (2 hash-changed, 1 not-after-last-block) 3
testnet.yml, official + contracts, no fault (42663085238) pass, matching state digests 0 0 0 – –

Comparison with the earlier hook experiment (s1r-delay, same hook):

  • baseline 42627663763: 12 re-signs; 298 untrusted marks, 125 of them on official nodes (248 bad signature, 44 bad pair, 6 silent); 3 of 12 re-entries excluded;
  • the same commits as this PR, run 42627654636: 51 resends, 0 marks, 3 of 51 excluded.

About the exclusions that remain. In the s1r-* runs, R5 and R25 come before the faults start at sequence 150, while rounds take about 0.5 s. In those rounds a re-entry costs a whole stage 1 (nextRound(true) also clears the stage-2 and stage-3 messages already received). The other confidants finish without that node, which happens in the baseline as well. With the fix, none of these exclusions comes with a stage-1/2 mark.

The fallback path. s1r-resign alters the stored stage 1 before each replay: the hash or the timestamp, alternately. This forces a new signature, so it shows the previous behaviour, and the bad pairs and exclusions are those of the old code. Receivers accepted every re-signed stage 1. A different, fuller package with another realTrusted mask cannot exercise this path: the mask does not enter the characteristic hash or the timestamp, so such a replay is still resent unchanged. Reaching hash-changed naturally needs a package that changes the last block.

fatal ConveyerBase> Meta round currently in conveyer. The line appears exactly once per replay, and only on n5: 2, 21 and 3 times in the runs above. It also appears in the baseline-hook run (12 for 12 replays), and never without the hook. It is the pre-existing Conveyer::setTable path of every same-round package replacement, and the node keeps running after it.

🤖 Generated with Claude Code

- the own signed stage 1 is stored before add_stage1(), which already runs the state's
  onStage1() handlers, so the store never depends on that order;
- a stored stamp that cannot be parsed is its own reason, "unreadable", instead of
  "not-after-last-block"; unit tests updated, reason names tested;
- docs: the two cases that still sign a second stage 1, as before (a writer-side replacement,
  which starts the round with updateRound = false, and a replacement inside a bootstrap round).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB merged commit 56ae42a into master Nov 9, 2026
9 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant